Image processor, 3d image capture device, image processing method, and image processing program

ABSTRACT

An image processor  7  includes: an in-focus area extracting section  72  that extracts an in-focus area of two images with parallax; a color conversion matrix calculating section  73  that obtains a color conversion matrix between the two images by reference to information about the colors of pixels that are included in the in-focus area of the two images; and a color conversion section  74  that converts the color of one of the two images by using the color conversion matrix.

TECHNICAL FIELD

The present application relates to a single-lens 3D image capturingtechnology for generating multiple images with parallax using a singleoptical system and a single image sensor.

BACKGROUND ART

Recently, the performance and functionality of digital cameras anddigital movie cameras that use some solid-state image sensor such as aCCD and a CMOS (which will be sometimes simply referred to herein as an“image sensor”) have been enhanced to an astonishing degree. Inparticular, the size of a pixel structure for use in a solid-state imagesensor has been further reduced these days thanks to rapid developmentof semiconductor device processing technologies, thus getting an evengreater number of pixels and drivers integrated together in asolid-state image sensor. As a result, the resolution of an image sensorhas lately increased rapidly from around one million pixels to tenmillion or more pixels in a matter of few years. On top of that, thequality of an image captured has also been improved significantly aswell.

As for display devices, on the other hand, LCD and plasma displays witha reduced depth now provide high-resolution and high-contrast images,thus realizing high performance without taking up too much space. Andsuch video quality improvement trends are now spreading from 2D imagesto 3D images. In fact, 3D display devices that achieve high imagequality, although they require the viewer to wear a pair of polarizationglasses, have been developed just recently.

As for the 3D image capturing technology, a typical 3D image capturedevice with a simple arrangement uses an image capturing system with twocameras to capture a right-eye image and a left-eye image. According tothe so-called “two-lens image capturing” technique, however, two camerasneed to be used, thus increasing not only the overall size of the imagecapture device but also the manufacturing cost as well. To overcome sucha problem, methods for capturing multiple images with parallax (whichwill be sometimes referred to herein as “multi-viewpoint images”) byusing a single camera have been researched and developed. Such a methodis called a “single-lens image capturing method”.

For example, Patent Document No. 1 discloses a technique for obtainingtwo images with parallax at the same time using color filters. FIG. 10schematically illustrates the image capturing system disclosed in PatentDocument No. 1. The image capturing system that uses that techniqueincludes a lens 3, a lens diaphragm 19, a light beam confining plate 20with two color filters 20 a and 20 b that have mutually differenttransmission wavelength ranges, and a photosensitive film 21. In thiscase, the color filters 20 a and 20 b may be filters that transmit red-and blue-based light rays, respectively.

In such an arrangement, the incoming light passes through the lens 3,the lens diaphragm 19 and the light beam confining plate 20 and producesan image on the photosensitive film. In the meantime, only red- andblue-based light rays are respectively transmitted through the two colorfilters 20 a and 20 b of the light beam confining plate 20. As a result,a magenta-based color image is produced on the photosensitive film bythe light rays that have been transmitted through the two color filters.In this case, since the color filters 20 a and 20 b are arranged atmutually different positions, the image produced on the photosensitivefilm comes to have parallax. Thus, if a photograph is developed with thephotosensitive film and viewed with a pair of glasses, in which red andblue films are attached to its right- and left-eye lenses, the viewercan view an image with depth. In this manner, according to the techniquedisclosed in Patent Document No. 1, multi-viewpoint images can beproduced using the two color filters.

According to the technique disclosed in Patent Document No. 1, the lightrays are imaged on the photosensitive film, thereby producing multipleimages with parallax there. Meanwhile, Patent Document No. 2 discloses atechnique for producing images with parallax by transforming incominglight into electrical signals. FIG. 11 schematically illustrates a lightbeam confining plate according to such a technique. Specifically,according to that technique, a light beam confining plate 22, which hasa red ray transmitting R area 22R, a green ray transmitting G area 22Gand a blue ray transmitting B area 22B, is arranged on a plane thatintersects with the optical axis of the imaging optical system at rightangles. And by getting the light rays that have been transmitted throughthose areas received by a color image sensor that has red-, green- andblue-ray-receiving R, G and B pixels, an image is generated based on thelight rays that have been transmitted through those areas.

Patent Document No. 3 also discloses a technique for obtaining imageswith parallax using a similar configuration to the one used in PatentDocument No. 2. FIG. 12 schematically illustrates a light beam confiningplate as disclosed in Patent Document No. 3. According to thattechnique, by making the incoming light pass through R, G and B areas23R, 23G and 23B of the light beam confining plate 23, multiple imageswith parallax can also be produced.

Patent Document No. 4 also discloses a technique for generating multipleimages with parallax using a pair of filters with mutually differentcolors, which are arranged symmetrically to each other with respect toan optical axis. By using red and blue filters as the pair of filters,an R pixel that senses a red ray observes the light that has beentransmitted through the red filter, while a B pixel that senses a blueray observes the light that has been transmitted through the bluefilter. Since the red and blue filters are arranged at two differentpositions, the light received by the R pixel and the light received bythe B pixel have come from mutually different directions. Consequently,the image observed by the R pixel and the image observed by the B pixelare ones viewed from two different viewpoints. By defining correspondingpoints between those images on a pixel-by-pixel basis, the magnitude ofparallax can be calculated. And based on the magnitude of parallaxcalculated and information about the focal length of the camera, thedistance from the camera to the subject can be obtained.

Patent Document No. 5 discloses a technique for obtaining informationabout a subject distance based on two images that have been generatedusing either a diaphragm to which two color filters with mutuallydifferent aperture sizes are attached or a diaphragm to which two colorfilters in two different colors are attached horizontally symmetricallywith respect to the optical axis. According to such a technique, iflight rays that have been transmitted through the red and blue colorfilters with mutually different aperture sizes are observed, the degreesof blur observed vary from one color to another. That is why the degreesof blur of the two images that are associated with the red and bluecolor filters vary according to the subject distance. By definingcorresponding points with respect to those images and comparing theirdegrees of blur to each other, information about the distance from thecamera to the subject can be obtained. On the other hand, if light raysthat have been transmitted through two color filters in two differentcolors that are attached horizontally symmetrically with respect to theoptical axis are observed, the direction from which the light observedhas come changes from one color to another. As a result, two images thatare associated with the red and blue color filters become images withparallax. And by defining corresponding points with respect to thoseimages and calculating the distance between those corresponding points,information about the distance from the camera to the subject can beobtained.

According to the techniques disclosed in Patent Documents Nos. 1 to 5mentioned above, images with parallax can be produced by arranging RGBcolor filters on a light beam confining plate. However, since the lightbeam confining plate is used, the percentage of the incoming light thatcan be used decreases significantly. In addition, increase the effect ofparallax, those RGB color filters should be arranged at distantpositions and should have decreased areas. In that case, however, thepercentage of the incoming light that can be used further decreases.

Unlike these techniques, Patent Document No. 6 discloses a technique forobtaining multiple images with parallax and a normal image that is freefrom the light quantity problem by using a diaphragm in which RGB colorfilters are arranged. According to that technique, when the diaphragm isclosed, only the light rays that have been transmitted through the RGBcolor filters are received. On the other hand, when the diaphragm isopened, the RGB color filter areas are outside of the optical path, andtherefore, the incoming light can be received entirely. Consequently,images with parallax can be obtained when the diaphragm is closed and anormal image that uses the incoming light highly efficiently can beobtained when the diaphragm is opened.

CITATION LIST Patent Literature

-   Patent Document No. 1: Japanese Laid-Open Patent Publication No.    2-171737-   Patent Document No. 2: Japanese Laid-Open Patent Publication No.    2002-344999-   Patent Document No. 3: Japanese Laid-Open Patent Publication No.    2009-276294-   Patent Document No. 4: Japanese Laid-Open Patent Publication No.    2010-38788-   Patent Document No. 5: Japanese Laid-Open Patent Publication No.    2010-79298-   Patent Document No. 6: Japanese Laid-Open Patent Publication No.    2003-134533

Non-Patent Literature

-   Non-Patent Document No. 1: “Image Segmentation Using Iterated Graph    Cuts Based on Multi-scale Smoothing”, Tomoyuki Nagahashi, Hironobu    Fujiyoshi, and Takeo Kanade, Transactions of Information Processing    Society of Japan CVIM, Vol. 1, No. 2, pp. 10-20, 2008.

SUMMARY OF INVENTION Technical Problem

According to these technologies of the related art, images with parallaxcan be certainly obtained, but the quantity of the light received by theimage sensor is much smaller than usual because primary color (RGB)based color filters are used. On the other hand, in order to use theincoming light sufficiently, a normal image that uses the incoming lighthighly efficiently needs to be obtained by using a mechanism thatremoves the color filter from the optical path by mechanical driving asdisclosed in Patent Document No. 6. If such a mechanism is provided,however, the overall size of the device increases too much and themanufacturing cost becomes too high.

To overcome these problems, an embodiment of the present inventionprovides an image capturing technique for obtaining multi-viewpointimages with the incoming light used highly efficiently without makingany mechanical driving.

Solution to Problem

An image processor as an embodiment of the present invention matches thecolors of two images with parallax to each other. The processorincludes: an in-focus area extracting section that extracts an in-focusarea of the two images; a color conversion matrix calculating sectionthat obtains a color conversion matrix between the two images byreference to information about the colors of pixels that are included inthe in-focus area of the two images; and a color conversion section thatconverts the color of one of the two images by using the colorconversion matrix.

This general and particular embodiment can be implemented as a system, amethod, a computer program or a combination thereof.

Advantageous Effects of Invention

According to an embodiment of the present invention, multi-viewpointimages can be obtained without making any mechanical driving and withthe light used more efficiently than ever.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 A block diagram illustrating an overall configuration for a 3Dimage capture device according to an embodiment.

FIG. 2 A schematic representation generally illustrating the relativearrangement of a light-transmitting plate, an optical system and animage sensor according to the embodiment.

FIG. 3 A view illustrating an arrangement of transmitting areas on alight-transmitting plate according to the embodiment.

FIG. 4 A view illustrating a basic arrangement of transmitting filtersin the image sensor according to the embodiment.

FIG. 5 A view illustrating a basic arrangement of transmitting filtersin the color image sensor according to the embodiment.

FIG. 6 Illustrates how the subject comes into, and out of, focusaccording to the embodiment.

FIG. 7A Shows an arrangement of functional blocks in an image signalgenerating section according to the embodiment.

FIG. 7B A flowchart showing the procedure of the color conversionprocessing to be carried out by the image signal generating sectionaccording to the embodiment.

FIG. 8 (a) shows one of the multi-viewpoint images according to theembodiment, (b) illustrates the high frequency components extracted, and(c) illustrates exemplary in-focus areas calculated.

FIG. 9 Illustrates conceptually how to perform the processing ofconverting, using a conversion matrix Mc, the colors of respectivepixels of an L image yet to be subjected to a color conversion.

FIG. 10 A view illustrating the arrangement of an image capturing systemaccording to a patent document.

FIG. 11 A view illustrating the appearance of a light beam confiningplate according to another patent document.

FIG. 12 A view illustrating the appearance of a light beam confiningplate according to still another patent document.

DESCRIPTION OF EMBODIMENTS

(1) To overcome the problems described above, an image processor as anembodiment of the present invention matches the colors of two imageswith parallax to each other, and includes: an in-focus area extractingsection that extracts an in-focus area of the two images; a colorconversion matrix calculating section that obtains a color conversionmatrix between the two images by reference to information about thecolors of pixels that are included in the in-focus area of the twoimages; and a color conversion section that converts the color of one ofthe two images by using the color conversion matrix.

(2) In one embodiment, the image processor of (1) further includes ahigh frequency component calculating section that calculates the highfrequency components of at least one of the two images. The in-focusarea extracting section extracts the in-focus areas based on the highfrequency components that have been calculated.

(3) In one embodiment of the image processor of (2), the in-focus areaextracting section extracts, as the in-focus area, the vicinity of highfrequency pixels in which the quantity of the high frequency componentsis greater than a predetermined threshold value.

(4) In one embodiment of the image processor of (3), the in-focus areaextracting section extracts, as the in-focus area, a rectangular areacomprised of n pixels×m pixels (where n and m are integers that areequal to or greater than one) including the high frequency pixels.

(5) In one embodiment of the image processor of (3) or (4), the in-focusarea extracting section extracts, as the in-focus area, a rectangulararea comprised of n pixels×m pixels (where n and m are integers that areequal to or greater than one) surrounding the high frequency pixels.

(6) In one embodiment of the image processor of one of (1) to (5), thecolor conversion matrix calculating section obtains the color conversionmatrix by linear computations by the minimum square method, the Mestimation method or the RAMSAC method.

(7) A 3D image capture device as an embodiment of the present inventionincludes: a light transmitting section that has two transmitting areaswith mutually different spectral transmittance characteristics; an imagesensor that is arranged to receive the light that has been transmittedthrough the light transmitting section and that includes two kinds ofpixels with mutually different spectral transmittance characteristics;and an image processing section that generates two images with parallaxbased on pixel signals supplied from the image sensor. The imageprocessing section includes: an in-focus area extracting section thatextracts an in-focus area of the two images; a color conversion matrixcalculating section that obtains a color conversion matrix between thetwo images by reference to information about the colors of pixels thatare included in the in-focus area of the two images; and a colorconversion section that converts the color of one of the two images byusing the color conversion matrix.

(8) An image processing method as an embodiment of the present inventionis designed to match the colors of two images with parallax to eachother, and includes the steps of: extracting an in-focus area of the twoimages; obtaining a color conversion matrix between the two images byreference to information about the colors of pixels that are included inthe in-focus area of the two images; and converting the color of one ofthe two images by using the color conversion matrix.

An image processing program as an embodiment of the present invention isdesigned to match the colors of two images with parallax to each other,and defined to make a computer perform the steps of: extracting anin-focus area of the two images; obtaining a color conversion matrixbetween the two images by reference to information about the colors ofpixels that are included in the in-focus area of the two images; andconverting the color of one of the two images by using the colorconversion matrix.

Hereinafter, more specific embodiments of the present invention will bedescribed with reference to the accompanying drawings. In the followingdescription, any element shown in multiple drawings and havingsubstantially the same function will be identified by the same referencenumeral. It should be noted that a signal or information representing animage will be sometimes referred to herein as just an “image”.

Embodiments

FIG. 1 is a block diagram illustrating an overall configuration for a 3Dimage capture device (which will be simply referred to herein as an“image capture device”) as an embodiment of the present invention. Theimage capture device of this embodiment is a digital electronic cameraand includes an image capturing section 100 and a signal processingsection 200 that receives a signal from the image capturing section 100and outputs a signal representing an image (i.e., an image signal).

The image capturing section 100 includes an image sensor 1 with a numberof photosensitive cells that are arranged on its image capturing plane,a light-transmitting plate 2 which has two transmitting areas withmutually different transmission wavelength ranges, an optical lens 3 forproducing an image on the image capturing plane of the image sensor 1,and an infrared cut filter 4. The image capturing section 100 furtherincludes a signal generating and receiving section 5, which not onlygenerates a fundamental signal to drive the image sensor 1 but alsoreceives the output signal of the image sensor 1 and sends it to thesignal processing section 200, and a sensor driving section 6 fordriving the image sensor 1 in accordance with the fundamental signalgenerated by the signal generating and receiving section 5. The imagesensor 1 is typically a CCD or CMOS sensor, which may be fabricated byknown semiconductor device processing technologies. The signalgenerating and receiving section 5 and the sensor driving section 30 maybe implemented as an LSI such as a CCD driver.

The signal processing section 200 includes an image signal generatingsection 7 for generating an image signal by processing the signalsupplied from the image capturing section 100, a memory 30 for storingvarious kinds of data for use to generate the image signal, and aninterface (I/F) section 8 for sending out the image signal thusgenerated to an external device. The image signal generating section 7may be a combination of a hardware component such as a known digitalsignal processor (DSP) and a software program for use to perform imageprocessing involving the image signal generation. The memory 30 may be aDRAM, for example. And the memory 30 not only stores the signal suppliedfrom the image capturing section 100 but also temporarily retains theimage data that has been generated by the image signal generatingsection 7 or compressed image data. These image data are then output toeither a storage medium or a display section (neither is shown) by wayof the interface section 8.

The image capture device of this embodiment actually further includes anelectronic shutter, a viewfinder, a power supply (or battery), aflashlight and other known components. However, description thereof willbe omitted herein because none of them are essential components thatwould make it difficult to understand how the present invention worksunless they were described in detail. Also, this configuration is onlyan example. Thus, in this embodiment, additional components other thanthe light-transmitting plate 2, the image sensor 1 and the image signalgenerating section 7 may be implemented as an appropriate combination ofknown elements.

Next, the configuration of the image capturing section 100 will bedescribed in further detail with reference to FIGS. 2 through 4.

FIG. 2 schematically illustrates the relative arrangement of thelight-transmitting plate 2, the lens 3 and the image sensor 1 in theimage capturing section 100. It should be noted that illustration of theother elements is omitted in FIG. 2. The lens 3 may be a lens unit thatis a group of lenses but is drawn in FIG. 2 as a single lens for thesake of simplicity.

The light-transmitting plate 2 includes two transmitting areas C1 andC2, of which the optical transmittances have mutually differentwavelength dependences (i.e., which have mutually different spectraltransmittances). The lens 3 is a known lens and condenses the light thathas been transmitted through the light-transmitting plate 2, therebyimaging the light on the image capturing plane 1 a of the image sensor1. In this embodiment, the rest of the light-transmitting plate 2 otherthan the transmitting areas C1 and C2 is made of an opaque member.

FIG. 3 is a front view of the light-transmitting plate 2 of thisembodiment. The light-transmitting plate 2, as well as the lens 3, has acircular shape in this embodiment but may also have any other shape. Ineach of the areas C1 and C2, arranged is a filter W1, W2 that transmitsa light ray falling within an arbitrary wavelength range included in thewavelength range of visible radiation. The transmittances of thesefilters W1 and W2 have mutually different wavelength dependences. Thatis to say, even if the same light has been transmitted through thesefilters, the light transmitted will have different brightness (orluminance) values, depending on whether the light has been transmittedthrough the area C1 or the area C2. As long as each filter W1, W2 hasthe function of transmitting the light at an intended transmittance, thefilter may be made of any material. For example, the filters may be madeof glass, plastic, cellophane or any other suitable material. Althoughthe filters W1 and W2 of this embodiment transmit light rays fallingwithin an arbitrary part of the wavelength range of visible radiation,the filters W1 and W2 do not always have to have such a characteristic.Optionally, one or both of the filters W1 and W2 may be configured notto transmit light falling within a part of the wavelength range ofvisible radiation.

These areas C1 and C2 are arranged with a certain gap left in the xdirection. The distance L between the respective centers of these areasis determined by the size of the lens 3 so that the image obtained willhave appropriate parallax, and may be set to be within the range of afew millimeters to several centimeters, for example. The areas C1 and C2are generally arranged horizontally symmetrically with respect to theoptical axis and have the same area. If such an arrangement is adopted,the quantities of the light rays to be incident on the areas C1 and C2can be substantially equal to each other. If multi-viewpoint images withvertical parallax need to be obtained depending on the intended use,then these areas C1 and C2 may be arranged vertically (i.e., in the ydirection).

Also, if the respective transmittances of the filters W1 and W2 arrangedin the areas C1 and C2 are significantly different from each other, thenthe values of photoelectrically converted signals (i.e., pixel values)obtained by the respective photosensitive cells of the image sensor 1(to be described later) will also be quite different. That is why theplanar areas of those areas C1 and C2 may be adjusted so that two imageswith parallax will have close brightness values. Alternatively, by usinga filter that evenly decreases the transmittance of every light rayfalling within the visible radiation range (such as an ND filter) alongwith the light-transmitting plate 2, the intensities of the light raytransmitted through these areas C1 and C2 may be adjusted to beapproximately equal to each other.

On the image capturing plane 1 a of the image sensor 1 shown in FIG. 2,there is an array of photosensitive cells that are arrangedtwo-dimensionally and an array of transmitting filters that are arrangedto face those photosensitive cells in the array. The array ofphotosensitive cells and the array of transmitting filters consist ofmultiple unit elements.

Each of those photosensitive cells is typically a photodiode, whichperforms photoelectric conversion and outputs an electrical signalrepresenting the quantity of the light received (which will be referredto herein as a “photoelectrically converted signal” or a “pixelsignal”). On the other hand, each transmitting filter may be made ofknown pigment or a stack of dielectric materials and is designed so asto transmit at least a part of the incoming light. In the followingdescription, the fundamental principle of this embodiment will bedescribed on the supposition that each unit element includes two typesof transmitting filters.

FIG. 4 is a top view schematically illustrating a portion of the arrayof transmitting filters according to this embodiment. As shown in FIG.4, a lot of transmitting filters 110 are arranged in columns and rows onthe image capturing plane 1 a. Each unit element is formed of twotransmitting filters 110 that are arranged close to each other and twophotosensitive cells 120 that face them. The two transmitting filters D1and D2 that are included in each unit element both transmit a light raywith an arbitrary wavelength falling within the visible radiationwavelength range but have mutually different spectral transmittances.That is to say, each of the transmitting filters D1 and D2 transmits alight ray falling within the red (R), green (G) or blue (B) wavelengthrange but their transmittances have mutually different wavelengthdependences. In the example illustrated in FIG. 4, two photosensitivecells 120 included in each unit element are arranged horizontally (i.e.,in the x direction). However, this is only an example of the presentinvention and the photosensitive cells may also be arranged in the imagesensor 1 in any other known pattern.

According to such an arrangement, the light that has entered this imagecapture device during an exposure process passes through thelight-transmitting plate 2, the lens 3, the infrared cut filter 4 andthe transmitting filters 110 and then is incident on the photosensitivecells 120. Each of those photosensitive cells 120 receives a light raythat has been transmitted through the area C1 or C2 of thelight-transmitting plate 2 and then through its associated transmittingfilter, and outputs a photoelectrically converted signal representingthe quantity of the light received. The photoelectrically convertedsignal that has been output from each photosensitive cell is sent to thesignal processing section 200 by way of the signal generating andreceiving section 5. In the signal processing section 200, the imagesignal generating section 7 generates images with parallax based on thesignals supplied from the image capturing section 100.

Hereinafter, the photoelectrically converted signals supplied from thosephotosensitive cells 120 will be described. Signals representing therespective intensities of light rays that have been transmitted throughthe areas C1 and C2 of the light-transmitting plate 2 will be identifiedherein by Ci1 and Ci2, respectively, with the subscript “i” attached. Inthis case, the rest of the incoming light other than the visibleradiation is supposed to have been cut. Also, according to thisembodiment, this incoming light is supposed to include light rays withevery wavelength falling within the visible radiation range in equalquantities. The spectral transmittance of the lens 3 and the infraredcut filter 4 combined will be identified herein by Tw. And the spectraltransmittances of the W1 and W2 filters of the areas C1 and C2 will beidentified herein by TC1 and TC2, respectively. Both of the filters W1and W2 transmit a light ray with an arbitrary wavelength falling withinthe visible radiation range but their transmittances vary according tothe wavelength. That is to say, although both of the filters W1 and W2transmit R, G and B rays, these color components are transmitted inmutually different percentages by the two filters. In the same way, thespectral transmittances of the transmitting filters D1 and D2 at theimage sensor 1 will be identified herein by TD1 and TD2, respectively.Just like TC1 and TC2, the transmittances TD1 and TD2 also varyaccording to the wavelength. But the transmitting filters have aproperty to transmit R, G and B rays. That is why according to thisembodiment, at least a part of the R, G and B components included in theincoming light is transmitted through all of those four filters C1, C2,D1 and D2. That is why each of the two photosensitive cells that facethe transmitting filters D1 and D2 of the image sensor 1 can obtain asignal in which the three color components of R, G and B are superposedone upon the other.

In this case, Tw, TC1, TC2, TD1 and TD2 are functions that depend on thewavelength λ of the incoming light. And the signals representing theintensities of light rays that have been transmitted through thetransmitting filters D1 and D2 and then received by photosensitive cellsthat face them are identified by d1 and d2, respectively. Furthermore,the integration operation of the spectral transmittances in the visibleradiation wavelength range will be identified herein by the sign Σ. Forexample, an integration operation ∫TwTC1TD1d λ with respect to thewavelength λ will be identified herein by Σ TwTC1TD1. In this case, theintegration is supposed to be performed in the entire visible radiationwavelength range. Then, d1 is proportional to the sum of Ci1 Σ TwTC1TD1and Ci2 Σ TwTC2TD2. Likewise, d2 is proportional to the sum of Ci1 ΣTwTC1TD2 and Ci2Σ TwTC2TD2. Supposing the constant of proportionalitywith respect to these relations is one, d1 and d2 can be represented bythe following Equations (1) and (2), respectively:

d1=Ci1 ΣTwTC1TD1+Ci2ΣTwTC2TD1  (1)

d2=Ci1ΣTwTC1TD2+Ci2ΣTwTC2TD2  (2)

Suppose, in Equations (1) and (2), Σ TwTC1TD1, Σ TwTC2TD1, Σ TwTC1TD2,and Σ TwTC2TD2 are identified by Mx11, Mx12, Mx21 and Mx22,respectively. Then, Equation (1) can be represented by the followingEquation (3) using a matrix:

$\begin{matrix}{\begin{pmatrix}{d\; 1} \\{d\; 2}\end{pmatrix} = {\begin{pmatrix}{M \times 11} & {M \times 12} \\{M \times 21} & {M \times 22}\end{pmatrix}\begin{pmatrix}{{Ci}\; 1} \\{{Ci}\; 2}\end{pmatrix}}} & (3)\end{matrix}$

Supposing the respective elements of an inverse matrix, which isobtained by inverting the matrix consisting of the elements Mx11 throughMx22 as represented by Equation (3), are identified by iM11 throughiM22, respectively, Equation (3) can be modified into the followingEquation (4). That is to say, the signals representing the intensitiesof the light rays that have been incident on the areas C1 and C2 can berepresented by using the photoelectrically converted signals d1 and d2:

$\begin{matrix}{\begin{pmatrix}{{Ci}\; 1} \\{{Ci}\; 2}\end{pmatrix} = {\begin{pmatrix}{{iM} \times 11} & {{iM} \times 12} \\{{iM} \times 21} & {{iM} \times 22}\end{pmatrix}\begin{pmatrix}{d\; 1} \\{d\; 2}\end{pmatrix}}} & (4)\end{matrix}$

By this Equation (4), signals representing the intensities of light raysthat have been transmitted through the areas C1 and C2 can be obtainedby using the pixel signals d1 and d2 generated by shooting an image.Since the areas C1 and C2 are spaced apart from each other in the xdirection, images produced based on the light rays that have beenincident on the areas C1 and C2, respectively, become two images viewedfrom different viewpoints. Consequently, those signals representing theintensities of light rays that have come from two different positionsand that have been generated as a result of the processing describedabove form multi-viewpoint images.

In order to generate multi-viewpoint images by such a method, pixelsignals need to be obtained from two photosensitive cells per unitelement. A condition for calculating multi-viewpoint images based thosepixel signals supplied from the two photosensitive cells is that thetransmitting filters D1 and D2 have mutually different spectraltransmittances in the visible radiation range. That is why according tothis embodiment, the configuration shown in FIG. 4 does not have to beadopted but a configuration shown in FIG. 5 in which each unit elementis comprised of four photosensitive cells and four transmitting filtersD1, R, G and B that are arranged to face them may also be adopted. Evenso, multi-viewpoint images can also be calculated in the same way aswill be described below.

The transmitting filters R, G and B shown in FIG. 5 are designed so asto respectively transmit red, green and blue components of incominglight. In this case, the red component refers to a light ray fallingwithin the range of approximately 600 to 700 nm. The green componentrefers to a light ray falling within the range of approximately 500 to600 nm. And the blue component refers to a light ray falling within therange of approximately 400 to 500 nm. However, this definition isadopted just for the sake of convenience. And it may be determinedappropriately which color component corresponds to what wavelengthrange. The transmitting filter D1 shown in FIG. 5 is the same as thefilter D1 shown in FIG. 4. The filter D1 may be a transparent filter,for example. And the sum of the pixel signals supplied from the threephotosensitive cells that face the transparent filters R, G and B isregarded as a single pixel signal d2. Thus, the pixel signal d1 suppliedfrom the photosensitive cell that faces the transparent filter D1 andthe pixel signal d2 include RGB color components in differentpercentages. In this manner, multi-viewpoint images can be calculated asin the method that has already been described with reference to FIG. 4.

The image sensor 1 shown in FIG. 5 further includes photosensitive cellsthat sense R, G and B rays, respectively (i.e., R, G and B pixels).Thus, a color image can be generated based on the color information thathas been obtained from these pixels. By adding color informationobtained from the R, G and B pixels to the luminance signals Ci1 and Ci2representing the multi-viewpoint images obtained by the method describedabove, color multi-viewpoint images can be obtained. Supposing theluminance signals representing the multi-viewpoint images are identifiedby Y1 (=Ci1) and Y2 (=Ci2), respectively, the signals obtained from theR and B pixels are identified by Rs and Bs, respectively, and the sum ofthe luminance signals Y1 and Y2 is YL=Y1+Y2, the color differencesignals are given by (YL−Rs) and (YL−Bs), respectively. After thesecolor difference signals have been generated, these color differencesignals are turned into low-frequency ones by a band-pass filter andthen superposed on the multi-viewpoint image signals Ci1 and Ci2,thereby obtaining color multi-viewpoint images.

In this case, the R, G and B values of the image representing the lightthat has been transmitted through the area C1 are identified by IMG(L)r,IMG(L)g and IMG(L)b, respectively, and the R, G and B values of theimage representing the light that has been transmitted through the areaC2 are identified by IMG(R)r, IMG(R)g and IMG(R)b, respectively. Then,these values can be calculated by the following Equations (5) and (6).In Equations (5) and (6), M represents a 3×3 conversion matrix fortransforming the luminance signal Y1 or Y2 and the two color differencesignals YL−Rs and YL−Bs into the RGB values of each image.

$\begin{matrix}{\begin{pmatrix}{{{IMG}(L)}r} \\{{{IMG}(L)}g} \\{{{IMG}(L)}b}\end{pmatrix} = {M\begin{pmatrix}{Y\; 1} \\{{YL} - {Rs}} \\{{YL} - {Bs}}\end{pmatrix}}} & (5) \\{\begin{pmatrix}{{{IMG}(R)}r} \\{{{IMG}(R)}g} \\{{{IMG}(R)}b}\end{pmatrix} = {M\begin{pmatrix}{Y\; 2} \\{{YL} - {Rs}} \\{{YL} - {Bs}}\end{pmatrix}}} & (6)\end{matrix}$

By performing the processing represented by Equations (5) and (6), colorimage signals can be obtained based on the luminance signals and thecolor difference signal. The image signal generating section 7calculates these image signals on a unit element basis, therebygenerating color multi-viewpoint images.

As can be seen, according to this method, both of the transmitting areasC1 and C2 provided for the light-transmitting plate 2 (or diaphragm)transmit at least partially a light ray falling within each of the R, Gand B wavelength ranges. That is why the output signals of the R, G andB pixels of the image sensor 1 include RGB information of the light thathas been transmitted through the area C1 and the RGB information of thelight that has been transmitted through the area C2. As a result, acolor difference signal can be calculated based on these pixel signals.

However, as the transmitting areas C1 and C2 provided for thelight-transmitting plate 2 have mutually different spectraltransmittances, the percentages of the R, G and B components included inthe light transmitted through the area C1 are different from those ofthe R, G and B components included in the light transmitted through thearea C2. That is why the luminance signals of multi-viewpoint images,which are signals representing the quantities of light transmitted, arealso different between the areas C1 and C2. As a result, the coloredmulti-viewpoint images also have different colors. And if a 3D image isgenerated using such two color-shifted images as left- and right-eyeimages, the 3D image will look unnatural to the viewer.

Thus, to avoid such a problem, the image signal generating section 7 ofthis embodiment obtains a conversion matrix representing the differencein color between the two colored multi-viewpoint images and makes colorcorrection so that the color of one of the two multi-viewpoint imagesmatches that of the other. As a result, the color difference between thetwo images can be reduced.

To obtain a conversion matrix for converting the color of one of twoimages viewed from mutually different viewpoints into that of the other,pixels should be associated with each other between the two images andthen a color conversion matrix should be obtained between the associatedpixels. In order to overcome such a problem, according to thisembodiment, attention is paid to the face that the magnitude of parallaxis zero at in-focus pixels of the multi-viewpoint images. In associatingpixels during an image matching process, no pixels are likely to beassociated with each other successfully in an image area where there areno edges. On the other hand, it is easy to find in-focus pixels. Forexample, as a general digital camera or digital movie usually has anauto-focus function, it is easy to get information about which area ofthe image is now in focus.

Once the correspondence between the pixels is known, the image signalgenerating section 7 gets the R, G and B values of those pixels, andobtains a matrix for converting the color of one of the twomulti-viewpoint images into that of the other based on the difference inR, G and B values between the associated pixels. After that, by usingthe conversion matrix thus obtained, the image signal generating section7 makes color conversion on one of the two multi-viewpoint imagesentirely. Hereinafter, this color conversion processing will bedescribed in detail.

A pair of associated pixels between two multi-viewpoint images referherein to two pixels representing the same point in a three-dimensionalspace in the reference one of the multi-viewpoint images and the otherimage. If two pixels represent the same point in a three-dimensionalspace, then those two pixels should naturally represent the same color.According to this embodiment, however, since the areas C1 and C2 of thelight-transmitting plate 2 have mutually different spectraltransmittances, the multi-viewpoint images that have been shot throughthese areas C1 and C2 ordinarily have different pixel values even if thelight has come from the same point in a three-dimensional space. That iswhy it is difficult to apply a normal matching method such as blockmatching to be adopted in a known stereoscopic method.

FIG. 6 illustrates, as an example, how incoming light is imaged.Specifically, FIG. 6 illustrates an exemplary situation where light thathas come from a subject 60 is incident on a pixel (x, y) of the imagesensor 1 through the light-transmitting plate 2 which is split into twoareas C1 and C2. In this example, (x, y) represents a pair ofcoordinates on the image capturing plane. FIG. 6( a) illustrates asituation where the focus is right on the subject 60, while FIG. 6( b)illustrates a situation where the focus is found before the subject 60.In the latter case, the subject 60 observed looks blurred. In this case,if two image signals Ci1 and Ci2 are calculated by making computationson the pixel (x, y) in the state shown in FIG. 6( a), both of the twoimage signals can be obtained as luminance signals representing thelight that has come from the same point P in a three-dimensional space.In the state shown in FIG. 6( b), on the other hand, the light that haspassed through the area C1 has come from a portion P3 of the subject 60in the three-dimensional space and the light that has passed through thearea C2 has come from another portion P2 of the subject 60 in thethree-dimensional space. Thus, in the situation shown in FIG. 6( b), thetwo image signals Ci1 and Ci2 do not represent the same point in thethree-dimensional space at the pixel (x, y) As can be seen, since theimage signals Ci1 and Ci2 represent the same point in athree-dimensional space at an in-focus pixel of an image, it can be saidthat such a pixel has no parallax. Consequently, it can be seen that inthe method of calculating multi-viewpoint images according to thisembodiment, the problem of searching for corresponding points isequivalent to finding in-focus pixels from an image.

In view of these considerations, in order to obtain a color conversionmatrix, first of all, multi-viewpoint images are searched for in-focuspixels according to this embodiment. Next, at each of those in-focuspixels, a color conversion matrix is obtained between the two images.And by converting the color of the reference one of the two images byusing the conversion matrix, color correction is made. In this manner,according to the method of this embodiment, a color conversion matrix isobtained from a limited number of in-focus pixels. Since an importantsubject in an image is often focused on generally speaking, thecorresponding point search method of this embodiment works fine.

according to this embodiment

FIG. 7A shows an arrangement of functional blocks in the image signalgenerating section 7 that performs the method described above. As shownin FIG. 7A, the image signal generating section 7 includes a highfrequency component calculating section 71, an in-focus area extractingsection 72, a color conversion matrix calculating section 73, and acolor conversion processing section 74. FIG. 7B is a flowchart showingthe procedure of the color conversion processing to be carried out bythe image signal generating section 7. Hereinafter, the respectiveprocessing steps will be described in detail.

First of all, in order to search the multi-viewpoint images for in-focuspixels, the image signal generating section 7 makes the high frequencycomponent calculating section 71 calculate high frequency components inthe image (in Step S1). FIG. 8 illustrates how to extract high frequencycomponents. Specifically, FIG. 8( a) shows one of the multi-viewpointimages and FIG. 8( b) illustrates the high frequency componentsextracted. In FIG. 8( b), the brighter a pixel is, the more highfrequency components the pixel includes. The high frequency componentcalculating section 71 detects a pixel, of which the high frequencycomponents have a quantity that is equal to or greater than apredetermined threshold value, as a “high frequency pixel”.

As a method for extracting high frequency components, pixels, of whichthe pixel values vary significantly in an image space, may be extractedby using some known edge extraction filter such as a sobel filter or aLaplacian filter. Alternatively, an image represented in a frequencyspace through a Fourier transform may be subjected to high-pass filterprocessing and then subjected to inverse Fourier transform. According tothis embodiment, any of these methods may be adopted as long as the highfrequency components of an image can be calculated.

Next, the image signal generating section 7 makes the in-focus areaextracting section 72 extract in-focus areas (in Step S2). The highfrequency pixels that have been detected by the high frequency componentcalculating section 71 represent the contour of the in-focus subject.Since an in-focus area is located in the vicinity of high frequencypixels, the in-focus area extracting section 72 of this embodimentextracts a rectangular area consisting of n pixels×m pixels (where n andm are integers that are equal to or greater than one) and including thehigh frequency pixels as an in-focus area. For example, a rectangulararea consisting of n pixels×m pixels that surround the high frequencypixels may be extracted as an in-focus area. If the subject image in thein-focus area includes a lot of high frequency components, n and m maybe set to be smaller values. Then, it is possible to prevent anout-of-focus portion (i.e., an out-of-focus area) from forming part ofthe in-focus area extracted with more certainty than a situation where nand m are set to be large values. On the other hand, if the subjectincludes a little high frequency components, then n and m are suitablyset to be large values in order to get a number of corresponding pointsthat is large enough to calculate the color conversion matrix. Toprevent such an out-of-focus area from forming part of the in-focus areaextracted, the center of the rectangular area does not have to be thecenter of the high frequency pixels but a rectangular area consisting ofn pixels×m pixels that extend from the high frequency pixels toward thecenter of the image may be used as the in-focus area. This method uses apriori knowledge that a subject to be focused on (i.e., a subject that ashooter pays attention to) often appears at the center of an image.Also, if it is known in advance, by using the face recognitiontechnology that is often adopted in recent digital cameras, that thein-focus contour is a face area, that face area may be used as thein-focus area as well.

The white frames shown in FIG. 8( c) indicate examples of in-focusareas. Since the magnitude of parallax is zero in the in-focus areas,the in-focus area calculating processing step described above may beperformed on only one of the two multi-viewpoint images. Such in-focusareas do not have to be rectangular areas but may also have any othershape such as polygonal, circular, or elliptical ones. In the exampleshown in FIG. 6( c), six in-focus areas are extracted. However, thenumber of in-focus areas to extract may also be determined arbitrarily.

Optionally, the in-focus area extracting section 72 may also extract anin-focus area based on the difference between the value of a pixelsurrounding the high frequency pixels and that of a pixel that is faraway from the high frequency pixels. For example, as in the methoddisclosed in Non-Patent Document No. 1, it may be determined, by thesegmentation method that uses the similarity between pixel values,whether a given pixel belongs to an in-focus area or not. According tothis method, based on color information of a background sampled andcolor information of the foreground, the degrees of likelihood (orsimilarity) of foreground and background are calculated. And by thegraph cut method that uses these degrees of similarity, the image isdivided into multiple areas to determine whether each pixel belongs tothe foreground or the background. That is to say, pixels which haveturned out to be similar to pixels near the high frequency pixels in thedivided areas may be regarded as belonging to the in-focus area.

Although an in-focus area is extracted according to this embodimentbased on the high frequency components of an image, there is no need tocalculate the high frequency components if the in-focus area can bedetected by another method. For example, in a special situation wherethe distance from the image capture device to the subject is known inadvance, the in-focus area may be detected based on that distance andthe focal length of the optical system.

Next, the image signal generating section 7 makes the color conversionmatrix calculating section 73 calculate a color conversion matrix (inStep S3). In the in-focus area that has been obtained by the in-focusarea extracting section 72, RGB values are obtained from each of the twomulti-viewpoint images and a color conversion matrix for converting theRGB values of one of the two images into those of the other image iscalculated. In this example, the two multi-viewpoint images areidentified by IMG(L) and IMG(R), respectively. The RGB values in thein-focus area of the one image IMG(L) are identified by IMG(L)r(i, j),IMG(L)g(i, j), and IMG(L)b(i, j), respectively. On the other hand, theRGB values of the other image IMG(R) are identified by IMG(R)r(i, j),IMG(R)g(i, j), and IMG(R)b(i, j), respectively. It should be noted thatthe subscripts i and j indicate the coordinates of a pixel in thein-focus area obtained by the in-focus area extracting section 72. Byusing these RGB values, a color conversion matrix for converting therespective color values of IMG(L) into those of IMG(R) may be obtainedby the following Equation (7):

$\begin{matrix}{{\begin{bmatrix}{{{IMG}(L)}{r\left( {i,j} \right)}} & {{{IMG}(L)}{g\left( {i,j} \right)}} & {{{IMG}(L)}{b\left( {i,j} \right)}} \\{{{IMG}(L)}{r\left( {{i + 1},j} \right)}} & {{{IMG}(L)}{g\left( {{i + 1},j} \right)}} & {{{IMG}(L)}{b\left( {{i + 1},j} \right)}} \\{{{IMG}(L)}{r\left( {{i + 2},j} \right)}} & {{{IMG}(L)}{g\left( {{i + 2},j} \right)}} & {{{IMG}(L)}{b\left( {{i + 2},j} \right)}} \\\ldots & \; & \;\end{bmatrix}{Mc}} = {\quad\begin{bmatrix}{{{IMG}(R)}{r\left( {i,j} \right)}} & {{{IMG}(R)}{g\left( {i,j} \right)}} & {{{IMG}(R)}{b\left( {i,j} \right)}} \\{{{IMG}(R)}{r\left( {{i + 1},j} \right)}} & {{{IMG}(R)}{g\left( {{i + 1},j} \right)}} & {{{IMG}(R)}{b\left( {{i + 1},j} \right)}} \\{{{IMG}(R)}{r\left( {{i + 2},j} \right)}} & {{{IMG}(R)}{g\left( {{i + 2},j} \right)}} & {{{IMG}(R)}{b\left( {{i + 2},j} \right)}} \\\ldots & \; & \;\end{bmatrix}}} & (7)\end{matrix}$

In Equation (7), the conversion matrix Mc is a 3×3 matrix. If the numberof pixels of the in-focus area is n′, then the RGB value matrices on theleft and right sides of Equation (7) become n′ 3 matrices. If n′ is lessthan three, no conversion matrix can be obtained. However, since asituation where the in-focus area is made up of only two pixels rarelyarises, the conversion matrix can be ordinarily obtained with noproblem. If n′≧3, the conversion matrix Mc may be obtained simply by theminimum square method. Speaking intuitively, according to the minimumsquare method, the conversion matrix Mc is obtained so as to minimizethe sum of squared errors between the product of the inverse matrix ofthe conversion matrix Mc to obtain and the matrix consisting of thecolor signals of IMG(R) and the matrix consisting of the color signalsof IMG(L). According to this method, if noise was included in the colorof IMG(R) or in the color of IMG(L), a conversion matrix that wouldminimize those errors should be obtained, and therefore, colorconversion could not be carried out properly in some cases. As it isgenerally not easy to find such noise, such a problem may be coped withby a robust statistic based method such as the M estimation method.According to the M estimation method, in obtaining the likelihood of aconversion matrix, not squared errors but the output value of apredetermined error function is used. The evaluation formulae of theminimum square method and the M estimation method are as follows:

minimum square method: minΣ∈²

M estimation method: minΣρ(∈)

As the function ρ for use in the M estimation method, used generally isa function, of which the output value increases as the error ∈decreases, and decreases as the error ∈ increases. Typical examples ofsuch functions include a German and McClure's ρ function. As a result,the influence of an element with a significant error diminishes, andtherefore, a conversion matrix which is robust against noise can beestimated.

The L Med S method is another robust statistic based method, and is alsocalled a “minimum center value method”, by which a conversion matrix isobtained so as to minimize the center value of errors. According to theL Med S method, a conversion matrix is obtained and the errors ∈ in thein-focus area are obtained on a pixel by pixel basis as in the methoddescribed above. The errors thus obtained are sorted and then a centervalue is obtained. For example, if the in-focus area is made up of 100pixels, 100 errors are obtained on a pixel by pixel basis from thein-focus area and sorted out, and then the 50^(th) error value isextracted. According to this method, if the noise is less than 0.50%,the error of the center value obtained from a proper conversion matrixis the error obtained from a pixel that is not affected by noise inprinciple. Consequently, the estimation can get done while being hardlyaffected by noise.

Also, if the RGB value distribution in the in-focus area were biased,then a conversion matrix could not be obtained as intended by theminimum square method. In that case, the conversion matrix may beestimated by another robust statistic based method called “RANSAC(random sample consensus) method”. According to this method, first ofall, three or more pixels in the in-focus area of multi-viewpoint imagesare sampled, thereby obtaining a color conversion matrix for convertingthe color of IMG(L) into that of IMG(R). If the color conversion matrixobtained is a proper one, the color of IMG(R) is converted into that ofIMG(L) by using the inverse matrix of the color conversion matrix onpixels that have not been sampled. If the magnitude of error becomesminimum when the color of the original IMG(L) is compared to that ofIMG(L) obtained by converting IMG(R), then the color conversion matrixobtained is regarded as a proper one. According to this method, if nonoise is included at a sample point from which the color conversionmatrix is obtained for the first time or if the color distribution isnot biased, then the color conversion matrix can be obtained properly.

Finally, the image signal generating section 7 makes the colorconversion processing section 74 perform the color conversion processingusing the color conversion matrix that has been obtained by the methoddescribed above (in Step S4). The color conversion can be carried out byconverting the RGB values IMG(L)r, IMG(L)g, and IMG(L)b of all pixels ofthe left-eye one of the multi-viewpoint images using the conversionmatrix Mc.

FIG. 9 illustrates conceptually how to perform the processing ofconverting, using a conversion matrix Mc, the colors of respectivepixels of the image on the left-hand side yet to be subjected to thecolor conversion. As shown in FIG. 9, by converting the entire left (L)image using the conversion matrix Mc that has been obtained by comparingthe L image to the right (R) image in an in-focus area, a colorconverted L image is generated. If the left image is replaced with thatcolor-converted L image, a left image, of which the color matches thatof the right image, can be obtained. As a result, a more natural 3Dimage can be generated.

As described above, the image capture device of this embodiment cangenerate multi-viewpoint images by using the light-transmitting plate 2having two transmitting areas with mutually different spectraltransmittances and the image sensor 1 having two or more kinds oftransmitting filters with mutually different spectral transmittances. Inparticular, the image signal generating section 7 of this embodimentobtains a color conversion matrix in an in-focus area of themulti-viewpoint images and corrects the color of one of the two imagesentirely by using the color conversion matrix. As a result, the colorsof the multi-viewpoint images can be matched to each other relativelyeasily.

In the embodiments described above, each of the filters W1 and W2arranged in the areas C1 and C2 of the light-transmitting plate 2 andthe filters D1 and D2 of the image sensor 1 has a property oftransmitting at least partially a light ray representing every colorcomponent of RGB. However, according to the present invention, suchfilters do not always have to be used. Even when filters that cut lightrays representing some color components are used, the color conversionprocessing of this embodiment can also be used.

In the embodiments described above, the light-transmitting plate 2 hasonly two transmitting areas C1 and C2 but may have three or moretransmitting areas. Even if the light-transmitting plate 2 has three ormore transmitting areas, two images associated with two arbitrarytransmitting areas can also have their colors matched to each other.Thus, the image processing of this embodiment can also be used no lesseffectively even in such a situation.

The image capture device according to the embodiments of the presentinvention generates an image signal by performing signal arithmeticoperations on a photoelectrically converted signal that has beenobtained by capturing an image. However, such processing of generatingan image signal by performing signal arithmetic operations may also becarried out by another device that is provided independently of thatimage capture device. For example, even if a signal that has beenobtained by an image capture device including the image capturingsection 100 of this embodiment is loaded into another device (imageprocessor) to get a program defining the image signal processingsection's (7) signal arithmetic processing described above executed by acomputer in that another device, the effects of the embodimentsdescribed above can also be achieved.

INDUSTRIAL APPLICABILITY

A 3D image capture device according to an embodiment of the presentinvention can be used effectively in any camera that ever uses asolid-state image sensor. Examples of those cameras include consumerelectronic cameras such as digital still cameras and digital camcordersand solid-state surveillance cameras for industrial use. Also, an imageprocessor according to the present invention can match the colors of twoimages that have parallax and that have been obtained by an imagecapturing system so that the positions of in-focus parts do not shiftfrom each other between two images. Thus, the image processor can beused to process an image signal that has been obtained by the imagecapturing system described above and input to a display device such a 3DTV set.

REFERENCE SIGNS LIST

-   1 solid-state image sensor-   1 a solid-state image sensor's image capturing plane-   2 light-transmitting plate-   2 a light transmitting section-   3 optical lens-   3 a optical element functioning as both light-transmitting plate and    optical lens-   4 infrared cut filter-   5 signal generating and receiving section-   6 sensor driving section-   7 image signal generating section-   8 interface section-   19 lens diaphragm-   20, 22, 23 light beam confining plate-   20 a color filter transmitting red-based light ray-   20 b color filter transmitting blue-based light ray-   21 photosensitive film-   22R, 23R R ray transmitting areas of light beam confining plate-   22G, 23G G ray transmitting areas of light beam confining plate-   22B, 23B B ray transmitting areas of light beam confining plate-   30 memory-   60 subject-   71 high frequency component calculating section-   72 in-focus area extracting section-   73 color conversion matrix calculating section-   74 color conversion processing section-   100 image capturing section-   110 transmitting filter-   120 photosensitive cell-   200 signal processing section

1. An image processor that matches the colors of two images withparallax to each other, the processor comprising: an in-focus areaextracting section that extracts an in-focus area of the two images; acolor conversion matrix calculating section that obtains a colorconversion matrix between the two images by reference to informationabout the colors of pixels that are included in the in-focus area of thetwo images; and a color conversion section that converts the color ofone of the two images by using the color conversion matrix.
 2. The imageprocessor of claim 1, further comprising a high frequency componentcalculating section that calculates the high frequency components of atleast one of the two images, wherein the in-focus area extractingsection extracts the in-focus areas based on the high frequencycomponents that have been calculated.
 3. The image processor of claim 2,wherein the in-focus area extracting section extracts, as the in-focusarea, the vicinity of high frequency pixels in which the quantity of thehigh frequency components is greater than a predetermined thresholdvalue.
 4. The image processor of claim 3, wherein the in-focus areaextracting section extracts, as the in-focus area, a rectangular areacomprised of n pixels×m pixels (where n and m are integers that areequal to or greater than one) including the high frequency pixels. 5.The image processor of claim 3, wherein the in-focus area extractingsection extracts, as the in-focus area, a rectangular area comprised ofn pixels×m pixels (where n and m are integers that are equal to orgreater than one) surrounding the high frequency pixels.
 6. The imageprocessor of claim 1, wherein the color conversion matrix calculatingsection obtains the color conversion matrix by linear computations bythe minimum square method, the M estimation method or the RAMSAC method.7. A 3D image capture device comprising: a light transmitting sectionthat has two transmitting areas with mutually different spectraltransmittance characteristics; an image sensor that is arranged toreceive the light that has been transmitted through the lighttransmitting section and that includes two kinds of pixels with mutuallydifferent spectral transmittance characteristics; and an imageprocessing section that generates two images with parallax based onpixel signals supplied from the image sensor, and wherein the imageprocessing section includes: an in-focus area extracting section thatextracts an in-focus area of the two images; a color conversion matrixcalculating section that obtains a color conversion matrix between thetwo images by reference to information about the colors of pixels thatare included in the in-focus area of the two images; and a colorconversion section that converts the color of one of the two images byusing the color conversion matrix.
 8. An image processing method formatching the colors of two images with parallax to each other, themethod comprising the steps of: extracting an in-focus area of the twoimages; obtaining a color conversion matrix between the two images byreference to information about the colors of pixels that are included inthe in-focus area of the two images; and converting the color of one ofthe two images by using the color conversion matrix.
 9. An imageprocessing program for matching the colors of two images with parallaxto each other, the program being defined to make a computer perform thesteps of: extracting an in-focus area of the two images; obtaining acolor conversion matrix between the two images by reference toinformation about the colors of pixels that are included in the in-focusarea of the two images; and converting the color of one of the twoimages by using the color conversion matrix.